{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dd6b9252-2470-4928-b7c1-a238f818c8c0",
   "metadata": {},
   "source": [
    "# Fine-Tune Llama 3.2-3B with MLX Distributed and LoRA\n",
    "\n",
    "This Notebook will show how to fine-tune Llama 3.2-3B on multiple GPUs and `mlx-lm`.\n",
    "\n",
    "The MLX runtime uses `mlx[cuda]` package to run distributed training on GPUs.\n",
    "\n",
    "MLX Distributed: https://ml-explore.github.io/mlx/build/html/usage/distributed.html\n",
    "\n",
    "MLX LM: https://github.com/ml-explore/mlx-lm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2101794-2700-499f-ab14-7798e8177d02",
   "metadata": {},
   "source": [
    "## Install the Kubeflow SDK\n",
    "\n",
    "You need to install the Kubeflow SDK to interact with Kubeflow Trainer APIs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59976ad2-0f80-4fc5-8fae-917fa0dc0841",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -U kubeflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8b85fb2-9572-4903-8cf1-930c4844aabd",
   "metadata": {},
   "source": [
    "## Create Script to Fine-Tune Llama 3.2\n",
    "\n",
    "We will use `mlx-lm` library to fine-tune Llama 3.2.\n",
    "\n",
    "`mlx-lm` is a Python package for generating text and fine-tuning LLMs with MLX.\n",
    "\n",
    "We will perform LoRA (Low-Rank Adaptation) fine-tuning to reduce number of trainable parameters and optimize GPU resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f626231c-3f49-40d6-bd37-6d8aff83cede",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-10-18T00:28:15.740719Z",
     "iopub.status.busy": "2025-10-18T00:28:15.740245Z",
     "iopub.status.idle": "2025-10-18T00:28:15.754837Z",
     "shell.execute_reply": "2025-10-18T00:28:15.754275Z",
     "shell.execute_reply.started": "2025-10-18T00:28:15.740685Z"
    }
   },
   "outputs": [],
   "source": [
    "def fine_tune_llama(hf_token: str, num_samples: str, batch_size: str):\n",
    "    import types\n",
    "    import os\n",
    "    import mlx.core as mx\n",
    "    from mlx_lm.lora import train_model, CONFIG_DEFAULTS\n",
    "    from mlx_lm.tuner.datasets import load_dataset\n",
    "    from mlx_lm.utils import load\n",
    "    from mlx_lm.generate import generate\n",
    "\n",
    "    os.environ[\"HF_TOKEN\"] = hf_token\n",
    "\n",
    "    # Set parameters for the mlx-lm.\n",
    "    args = types.SimpleNamespace()\n",
    "    args.model = \"meta-llama/Llama-3.2-3B-Instruct\"\n",
    "    args.data = \"mlx-community/WikiSQL\"\n",
    "    args.train = True\n",
    "    # Configure LoRA settings to reduce number of trainable params.\n",
    "    args.lora_parameters = {\n",
    "        \"rank\": 8,\n",
    "        \"dropout\": 0.05,\n",
    "        \"scale\": 20.0,\n",
    "    }\n",
    "\n",
    "    args.iters = int(num_samples)\n",
    "    args.batch_size = int(batch_size)\n",
    "\n",
    "    # Set defaults for other required parameters\n",
    "    for k, v in CONFIG_DEFAULTS.items():\n",
    "        if not hasattr(args, k):\n",
    "            setattr(args, k, v)\n",
    "\n",
    "    model, tokenizer = load(args.model)\n",
    "    train_set, valid_set, test_set = load_dataset(args, tokenizer)\n",
    "\n",
    "    # Start the Llama distributed fine-tuning.\n",
    "    train_model(args, model, train_set, valid_set)\n",
    "\n",
    "    # Evaluate the fine-tuned adapter.\n",
    "    dist = mx.distributed.init(strict=True, backend=\"mpi\")\n",
    "    if dist.rank() == 0:\n",
    "        print(\"=\" * 100)\n",
    "        print(f\"Training is complete. Adapters saved to: {args.adapter_path}\")\n",
    "        print(\"Evaluate the fine-tuned LoRA adapter for Llama 3.2\")\n",
    "\n",
    "        finetuned_model, finetuned_tokenizer = load(\n",
    "            args.model, adapter_path=args.adapter_path\n",
    "        )\n",
    "\n",
    "        # Generate response using the fine-tuned adapter.\n",
    "        sample_prompt = \"What is SQL?\"\n",
    "\n",
    "        print(f\"Prompt: {sample_prompt}\")\n",
    "        print(\"Response:\")\n",
    "\n",
    "        response = generate(\n",
    "            model=finetuned_model,\n",
    "            tokenizer=finetuned_tokenizer,\n",
    "            prompt=sample_prompt,\n",
    "            max_tokens=1000,\n",
    "            verbose=False,\n",
    "        )\n",
    "\n",
    "        print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48096ce0-33f3-4249-b1cb-b83e5ee9d258",
   "metadata": {},
   "source": [
    "## Get the MLX Runtime\n",
    "\n",
    "You can list the available Kubeflow Trainer runtimes with the `list_runtimes()` API.\n",
    "\n",
    "The name of the MLX runtime is `mlx-distributed`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "be6a71c0-d61e-442a-93e7-211d93942de3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-10-18T00:28:17.127390Z",
     "iopub.status.busy": "2025-10-18T00:28:17.126903Z",
     "iopub.status.idle": "2025-10-18T00:28:18.853191Z",
     "shell.execute_reply": "2025-10-18T00:28:18.852050Z",
     "shell.execute_reply.started": "2025-10-18T00:28:17.127361Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: mlx-distributed, Framework: mlx, Trainer Type: CustomTrainer\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from kubeflow.trainer import TrainerClient, CustomTrainer\n",
    "\n",
    "for r in TrainerClient().list_runtimes():\n",
    "    if r.name == \"mlx-distributed\":\n",
    "        print(f\"Name: {r.name}, Framework: {r.trainer.framework}, Trainer Type: {r.trainer.trainer_type.value}\\n\")\n",
    "        mlx_runtime = r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbe68e70-74ee-4c38-ae3e-8c5b214e3e91",
   "metadata": {},
   "source": [
    "## Get the Runtime Packages\n",
    "\n",
    "You can see the available Python packages and GPUs with the `get_runtime_packages()` API.\n",
    "\n",
    "The API shows available GPUs with CUDA driver on the single training node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bf65074b-7170-4a14-a9f8-3ea9420f090e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-10-18T00:31:35.896872Z",
     "iopub.status.busy": "2025-10-18T00:31:35.896352Z",
     "iopub.status.idle": "2025-10-18T00:32:03.235146Z",
     "shell.execute_reply": "2025-10-18T00:32:03.233613Z",
     "shell.execute_reply.started": "2025-10-18T00:31:35.896822Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: Permanently added '[t9ffdbb239d2-node-0-0.t9ffdbb239d2]:2222' (ECDSA) to the list of known hosts.\n",
      "Python: 3.10.12 (main, Aug 15 2025, 14:32:43) [GCC 11.4.0]\n",
      "Package                Version\n",
      "---------------------- -----------\n",
      "aiohappyeyeballs       2.6.1\n",
      "aiohttp                3.13.1\n",
      "aiosignal              1.4.0\n",
      "async-timeout          5.0.1\n",
      "attrs                  25.4.0\n",
      "certifi                2025.10.5\n",
      "charset-normalizer     3.4.4\n",
      "datasets               4.0.0\n",
      "dill                   0.3.8\n",
      "filelock               3.20.0\n",
      "frozenlist             1.8.0\n",
      "fsspec                 2025.3.0\n",
      "hf-xet                 1.1.10\n",
      "huggingface-hub        0.35.3\n",
      "idna                   3.11\n",
      "Jinja2                 3.1.6\n",
      "MarkupSafe             3.0.3\n",
      "mlx                    0.28.0\n",
      "mlx-cuda               0.28.0\n",
      "mlx-data               0.1.0\n",
      "mlx-lm                 0.26.3\n",
      "multidict              6.7.0\n",
      "multiprocess           0.70.16\n",
      "numpy                  2.2.6\n",
      "nvidia-cublas-cu12     12.9.1.4\n",
      "nvidia-cuda-nvrtc-cu12 12.9.86\n",
      "nvidia-cudnn-cu12      9.14.0.64\n",
      "packaging              25.0\n",
      "pandas                 2.3.3\n",
      "pip                    22.0.2\n",
      "propcache              0.4.1\n",
      "protobuf               6.33.0\n",
      "pyarrow                21.0.0\n",
      "python-dateutil        2.9.0.post0\n",
      "pytz                   2025.2\n",
      "PyYAML                 6.0.3\n",
      "regex                  2025.9.18\n",
      "requests               2.32.5\n",
      "safetensors            0.6.2\n",
      "setuptools             59.6.0\n",
      "six                    1.17.0\n",
      "tokenizers             0.22.1\n",
      "tqdm                   4.67.1\n",
      "transformers           4.57.1\n",
      "typing_extensions      4.15.0\n",
      "tzdata                 2025.2\n",
      "urllib3                2.5.0\n",
      "wheel                  0.37.1\n",
      "xxhash                 3.6.0\n",
      "yarl                   1.22.0\n",
      "\n",
      "Available GPUs on the single training node\n",
      "Sat Oct 18 00:31:57 2025       \n",
      "+-----------------------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 550.163.01             Driver Version: 550.163.01     CUDA Version: 12.8     |\n",
      "|-----------------------------------------+------------------------+----------------------+\n",
      "| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |\n",
      "|                                         |                        |               MIG M. |\n",
      "|=========================================+========================+======================|\n",
      "|   0  NVIDIA A100-SXM4-40GB          On  |   00000000:90:1D.0 Off |                    0 |\n",
      "| N/A   49C    P0             58W /  400W |       1MiB /  40960MiB |      0%      Default |\n",
      "|                                         |                        |             Disabled |\n",
      "+-----------------------------------------+------------------------+----------------------+\n",
      "                                                                                         \n",
      "+-----------------------------------------------------------------------------------------+\n",
      "| Processes:                                                                              |\n",
      "|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |\n",
      "|        ID   ID                                                               Usage      |\n",
      "|=========================================================================================|\n",
      "|  No running processes found                                                             |\n",
      "+-----------------------------------------------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "TrainerClient().get_runtime_packages(mlx_runtime)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b54ed60e-21bb-4184-963a-c1c690da1f1e",
   "metadata": {},
   "source": [
    "## Create TrainJob with MLX Distributed\n",
    "\n",
    "Use the `train()` API to create distributed TrainJob on **4 GPUs**. Every MPI training node uses 1 GPU.\n",
    "\n",
    "**Note** Update the HF Token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "32a30359-d275-4e4a-b714-da9dae5d5358",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-10-18T00:32:27.536464Z",
     "iopub.status.busy": "2025-10-18T00:32:27.535867Z",
     "iopub.status.idle": "2025-10-18T00:32:27.541831Z",
     "shell.execute_reply": "2025-10-18T00:32:27.540820Z",
     "shell.execute_reply.started": "2025-10-18T00:32:27.536388Z"
    }
   },
   "outputs": [],
   "source": [
    "# HF_TOKEN = \"hf_add_your_token\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c87e3d06-1cbf-4e14-acad-10a7dc2292a7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-10-18T00:32:44.867883Z",
     "iopub.status.busy": "2025-10-18T00:32:44.867247Z",
     "iopub.status.idle": "2025-10-18T00:32:46.099166Z",
     "shell.execute_reply": "2025-10-18T00:32:46.098089Z",
     "shell.execute_reply.started": "2025-10-18T00:32:44.867837Z"
    }
   },
   "outputs": [],
   "source": [
    "job_id = TrainerClient().train(\n",
    "    trainer=CustomTrainer(\n",
    "        func=fine_tune_llama,\n",
    "        func_args={\n",
    "            \"hf_token\": HF_TOKEN,\n",
    "            \"num_samples\": \"100\",\n",
    "            # Batch size must be divisible by the number of GPUs. (8 / 4 = 2) per training node.\n",
    "            \"batch_size\": \"8\",\n",
    "        },\n",
    "        num_nodes=4,  # Fine-Tune Llama3.2 on 4 GPUs.\n",
    "        resources_per_node={\n",
    "            \"gpu\": 1\n",
    "        },\n",
    "    ),\n",
    "    runtime=mlx_runtime,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9c526b36-3772-4512-b2e4-b794d600b384",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-10-18T00:32:46.665271Z",
     "iopub.status.busy": "2025-10-18T00:32:46.664585Z",
     "iopub.status.idle": "2025-10-18T00:32:46.675336Z",
     "shell.execute_reply": "2025-10-18T00:32:46.674430Z",
     "shell.execute_reply.started": "2025-10-18T00:32:46.665188Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ja1595e2b2a3'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train API generates a random TrainJob id.\n",
    "job_id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6952099-0e24-4f48-9520-10cc90b5d0b9",
   "metadata": {},
   "source": [
    "## Check the TrainJob Info\n",
    "\n",
    "Use the `list_jobs()` and `get_job()` APIs to get information about created TrainJob and its steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "05477438-0e23-45e3-92f3-76cfeff5a31c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-10-18T00:32:48.569555Z",
     "iopub.status.busy": "2025-10-18T00:32:48.569202Z",
     "iopub.status.idle": "2025-10-18T00:32:50.301497Z",
     "shell.execute_reply": "2025-10-18T00:32:50.300759Z",
     "shell.execute_reply.started": "2025-10-18T00:32:48.569537Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TrainJob: ja1595e2b2a3, Status: Created, Created at: 2025-10-18 00:32:46+00:00\n"
     ]
    }
   ],
   "source": [
    "for job in TrainerClient().list_jobs():\n",
    "    print(f\"TrainJob: {job.name}, Status: {job.status}, Created at: {job.creation_timestamp}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d0af3dd0-ae67-4b77-9154-ce46326b727f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-10-18T00:33:07.890737Z",
     "iopub.status.busy": "2025-10-18T00:33:07.889994Z",
     "iopub.status.idle": "2025-10-18T00:33:09.651553Z",
     "shell.execute_reply": "2025-10-18T00:33:09.650804Z",
     "shell.execute_reply.started": "2025-10-18T00:33:07.890666Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: node-0, Status: Running, Devices: gpu x 1\n",
      "\n",
      "Step: node-1, Status: Running, Devices: gpu x 1\n",
      "\n",
      "Step: node-2, Status: Running, Devices: gpu x 1\n",
      "\n",
      "Step: node-3, Status: Running, Devices: gpu x 1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# We execute mpirun command on node-0, which functions as the MPI Launcher node.\n",
    "for c in TrainerClient().get_job(name=job_id).steps:\n",
    "    print(f\"Step: {c.name}, Status: {c.status}, Devices: {c.device} x {c.device_count}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc0e85af-9dab-4517-944f-72c6625ed0c9",
   "metadata": {},
   "source": [
    "## Get the TrainJob Logs\n",
    "\n",
    "Use the `get_job_logs()` API to retrieve the TrainJob logs.\n",
    "\n",
    "The fine-tuning runs on 4 training nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4be42866-b688-427d-a9a3-fe8e0aff5c9b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-10-18T00:33:36.121463Z",
     "iopub.status.busy": "2025-10-18T00:33:36.120954Z",
     "iopub.status.idle": "2025-10-18T00:35:02.313901Z",
     "shell.execute_reply": "2025-10-18T00:35:02.312924Z",
     "shell.execute_reply.started": "2025-10-18T00:33:36.121426Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: Permanently added '[ja1595e2b2a3-node-0-0.ja1595e2b2a3]:2222' (ECDSA) to the list of known hosts.\n",
      "Warning: Permanently added '[ja1595e2b2a3-node-0-1.ja1595e2b2a3]:2222' (ECDSA) to the list of known hosts.\n",
      "Warning: Permanently added '[ja1595e2b2a3-node-0-2.ja1595e2b2a3]:2222' (ECDSA) to the list of known hosts.\n",
      "Fetching 11 files: 100%|██████████| 11/11 [00:08<00:00,  1.34it/s]\n",
      "Fetching 11 files: 100%|██████████| 11/11 [00:08<00:00,  1.30it/s]\n",
      "Fetching 11 files: 100%|██████████| 11/11 [00:08<00:00,  1.25it/s]\n",
      "Fetching 11 files: 100%|██████████| 11/11 [00:08<00:00,  1.23it/s]\n",
      "Loading Hugging Face dataset mlx-community/WikiSQL.\n",
      "Loading Hugging Face dataset mlx-community/WikiSQL.\n",
      "Loading Hugging Face dataset mlx-community/WikiSQL.\n",
      "Loading Hugging Face dataset mlx-community/WikiSQL.\n",
      "Generating train split: 100%|██████████| 1000/1000 [00:00<00:00, 285307.39 examples/s]\n",
      "Generating valid split: 100%|██████████| 100/100 [00:00<00:00, 62110.23 examples/s]\n",
      "Generating test split: 100%|██████████| 100/100 [00:00<00:00, 90336.08 examples/s]\n",
      "Generating train split: 100%|██████████| 1000/1000 [00:00<00:00, 304752.16 examples/s]\n",
      "Generating valid split: 100%|██████████| 100/100 [00:00<00:00, 81049.35 examples/s]\n",
      "Generating test split: 100%|██████████| 100/100 [00:00<00:00, 66798.92 examples/s]\n",
      "Trainable parameters: 0.041% (1.311M/3212.750M)\n",
      "Starting training..., iters: 100\n",
      "Trainable parameters: 0.041% (1.311M/3212.750M)\n",
      "Starting training..., iters: 100\n",
      "Generating train split: 100%|██████████| 1000/1000 [00:00<00:00, 296920.86 examples/s]\n",
      "Generating valid split: 100%|██████████| 100/100 [00:00<00:00, 62798.38 examples/s]\n",
      "Generating test split: 100%|██████████| 100/100 [00:00<00:00, 88375.56 examples/s]\n",
      "Trainable parameters: 0.041% (1.311M/3212.750M)\n",
      "Starting training..., iters: 100\n",
      "Generating train split: 100%|██████████| 1000/1000 [00:00<00:00, 296710.81 examples/s]\n",
      "Generating valid split: 100%|██████████| 100/100 [00:00<00:00, 63138.70 examples/s]\n",
      "Generating test split: 100%|██████████| 100/100 [00:00<00:00, 87783.68 examples/s]\n",
      "Trainable parameters: 0.041% (1.311M/3212.750M)\n",
      "Starting training..., iters: 100\n",
      "Node 0 of 4\n",
      "Node 1 of 4\n",
      "Node 3 of 4\n",
      "Node 2 of 4\n",
      "Calculating loss...: 100%|██████████| 12/12 [00:09<00:00,  1.23it/s]\n",
      "Calculating loss...: 100%|██████████| 12/12 [00:09<00:00,  1.22it/s]\n",
      "Calculating loss...: 100%|██████████| 12/12 [00:09<00:00,  1.22it/s]\n",
      "Calculating loss...: 100%|██████████| 12/12 [00:10<00:00,  1.18it/s]\n",
      "Iter 1: Val loss 2.928, Val took 10.137s\n",
      "Iter 10: Train loss 2.745, Learning Rate 1.000e-05, It/sec 0.150, Tokens/sec 94.791, Trained Tokens 6308, Peak mem 8.137 GB\n",
      "Iter 20: Train loss 2.031, Learning Rate 1.000e-05, It/sec 24.480, Tokens/sec 15610.806, Trained Tokens 12685, Peak mem 8.137 GB\n",
      "Iter 30: Train loss 1.694, Learning Rate 1.000e-05, It/sec 9.703, Tokens/sec 6380.548, Trained Tokens 19261, Peak mem 8.137 GB\n",
      "Iter 40: Train loss 1.596, Learning Rate 1.000e-05, It/sec 20.303, Tokens/sec 12813.433, Trained Tokens 25572, Peak mem 8.137 GB\n",
      "Iter 50: Train loss 1.577, Learning Rate 1.000e-05, It/sec 21.041, Tokens/sec 13500.092, Trained Tokens 31988, Peak mem 8.137 GB\n",
      "Iter 60: Train loss 1.505, Learning Rate 1.000e-05, It/sec 23.805, Tokens/sec 15299.642, Trained Tokens 38415, Peak mem 8.137 GB\n",
      "Iter 70: Train loss 1.417, Learning Rate 1.000e-05, It/sec 24.097, Tokens/sec 15612.534, Trained Tokens 44894, Peak mem 8.137 GB\n",
      "Iter 80: Train loss 1.436, Learning Rate 1.000e-05, It/sec 17.106, Tokens/sec 10993.755, Trained Tokens 51321, Peak mem 8.137 GB\n",
      "Iter 90: Train loss 1.495, Learning Rate 1.000e-05, It/sec 11.841, Tokens/sec 7768.611, Trained Tokens 57882, Peak mem 8.573 GB\n",
      "Calculating loss...: 100%|██████████| 12/12 [00:00<00:00, 58.48it/s]\n",
      "Calculating loss...: 100%|██████████| 12/12 [00:00<00:00, 58.15it/s]\n",
      "Calculating loss...: 100%|██████████| 12/12 [00:00<00:00, 57.79it/s]\n",
      "Calculating loss...: 100%|██████████| 12/12 [00:00<00:00, 57.07it/s]\n",
      "Iter 100: Val loss 1.468, Val took 0.216s\n",
      "Iter 100: Train loss 1.387, Learning Rate 1.000e-05, It/sec 23.440, Tokens/sec 14769.735, Trained Tokens 64183, Peak mem 8.573 GB\n",
      "Iter 100: Saved adapter weights to adapters/adapters.safetensors and adapters/0000100_adapters.safetensors.\n",
      "Saved final weights to adapters/adapters.safetensors.\n",
      "====================================================================================================\n",
      "Training is complete. Adapters saved to: adapters\n",
      "Evaluate the fine-tuned LoRA adapter for Llama 3.2\n",
      "Fetching 11 files: 100%|██████████| 11/11 [00:00<00:00, 41753.25it/s]\n",
      "Prompt: What is SQL?\n",
      "Response:\n",
      "SQL (Structured Query Language) is a programming language designed for managing and manipulating data in relational database management systems (RDBMS). It is used to perform various operations such as creating, modifying, and querying databases.\n",
      "SQL is a declarative language, meaning that you specify what you want to do with your data, rather than how to do it. This makes it easier to learn and use, especially for those who are not experienced programmers.\n",
      "Some common SQL operations include:\n",
      "Creating a new table: `CREATE TABLE`\n",
      "Inserting data into a table: `INSERT INTO`\n",
      "Selecting data from a table: `SELECT`\n",
      "Updating data in a table: `UPDATE`\n",
      "Deleting data from a table: `DELETE`\n",
      "SQL is used in a wide range of applications, including web development, business intelligence, and data analysis. It is a fundamental skill for anyone working with databases, and is widely used in many industries.\n",
      "\n",
      "Here is an example of a simple SQL query:\n",
      "```sql\n",
      "SELECT * FROM customers WHERE country = 'USA';\n",
      "```\n",
      "This query selects all columns (`*`) from the `customers` table where the `country` column is equal to `'USA'`.\n",
      "\n",
      "SQL has many variations, including:\n",
      "\n",
      "* MySQL\n",
      "* PostgreSQL\n",
      "* Microsoft SQL Server\n",
      "* Oracle\n",
      "\n",
      "Each of these variations has its own set of features and syntax, but they all share the same basic structure and functionality.\n",
      "\n",
      "I hope this helps! Let me know if you have any questions or need further clarification.\n"
     ]
    }
   ],
   "source": [
    "for logline in TrainerClient().get_job_logs(job_id, follow=True):\n",
    "    print(logline)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bff39226-e207-4daa-87e7-60ac4dd5869d",
   "metadata": {},
   "source": [
    "## Delete the TrainJob\n",
    "\n",
    "When TrainJob is finished, you can delete the resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53978afb-d4e8-4e06-91af-fe6e3538d211",
   "metadata": {},
   "outputs": [],
   "source": [
    "TrainerClient().delete_job(job_id)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
